Objectives

This notebook will demonstrate how to:

  • Identify when Gene Set Variation Analysis (GSVA) is well-suited for an analysis
  • Perform GSVA on transformed RNA-seq data with the GSVA package
  • Generate random gene sets using functional programming tools from the purrr package

So far every pathway analysis method we’ve covered relies on some information about groups of samples in our data. For over-representation analysis (ORA), we created gene sets from two different two group comparisons. In the Gene Set Enrichment Analysis (GSEA) example, we used statistics from a differential gene expression (DGE) analysis where we compared MYCN amplified cell lines to non-amplified cell lines; we needed that amplification status information.

What if we’re less sure about groups in our data or we want to analyze our data in a more unsupervised manner?

In this notebook we will cover a method called Gene Set Variation Analysis (GSVA) (Hänzelmann et al. 2013) that allows us to calculate gene set or pathway scores on a per-sample basis.

We like this quote from the GSVA paper (Hänzelmann et al. 2013) to set the stage:

While [gene set enrichment] methods are generally regarded as end points of a bioinformatic analysis, GSVA constitutes a starting point to build pathway-centric models of biology.

Rather than contextualizing some results you already have from another analysis like DGE, GSVA is designed to provide an estimate of pathway variation for each of the samples in an experiment. Note that these scores will depend on the samples included in the dataset when you run GSVA; if you added more samples and reran GSVA, you would expect the scores to change.

Set up

Libraries

# Pipes
library(magrittr)
# Gene Set Variation Analysis
library(GSVA)

Directories and files

Directories

# We have some medulloblastoma data from the OpenPBTA project that we've 
# prepared ahead of time
input_dir <- file.path("data", "open-pbta")

# Create a directory specifically for the results using this dataset
output_dir <- file.path("results", "open-pbta") 
if (!dir.exists(output_dir)) {
  dir.create(output_dir, recursive = TRUE)
}

Input

We have VST transformed RNA-seq data, annotated with gene symbols, that has been collapsed such that there are no duplicated gene identifiers (see setup).

rnaseq_file <- file.path(input_dir, "medulloblastoma_vst_collapsed.tsv")

Output

gsva_results_file <- file.path(output_dir, "medulloblastoma_gsva_results.tsv")

Gene sets

The function that we will use to run GSVA wants the gene sets to be in a list, rather than a tidy data frame that we used with clusterProfiler (although it does accept other formats).

We’re going to take this opportunity to introduce a different format that gene sets are often distributed in called GMT (Gene Matrix Transposed).

We’re going to read in the Hallmark collection file directly from MSigDB, rather than using msigdbr like we did in earlier notebooks.

The RNA-seq data uses gene symbols, so we need gene sets that use gene symbols, too.

# R can often read in data from a URL
hallmarks_url <- "https://data.broadinstitute.org/gsea-msigdb/msigdb/release/7.1/h.all.v7.1.symbols.gmt"

# QuSAGE is another pathway analysis method, the qusage package has a function
# for reading GMT files and turning them into a list
hallmarks_list <- qusage::read.gmt(hallmarks_url)

What does this list look like?

head(hallmarks_list)

RNA-seq data

We have VST transformed RNA-seq data, which is on a log2-like scale. These data are from the Open Pediatric Brain Tumor Atlas (OpenPBTA) OpenPBTA is a collaborative project organized by the CCDL and the Center for Data-Driven Discovery in Biomedicine (D3b) at the Children’s Hospital of Philadelphia conducted openly on GitHub.

You can read more about the project here.

We’re only working with the medulloblastoma samples in this example.

rnaseq_df <- readr::read_tsv(rnaseq_file)

── Column specification ───────────────────────────────────────────────────────────────────────────────────────────
cols(
  .default = col_double(),
  gene_symbol = col_character()
)
ℹ Use `spec()` for the full column specifications.
# What does the RNA-seq data frame look like?
rnaseq_df[1:5, 1:5]

For GSVA, we need a matrix.

rnaseq_mat <- rnaseq_df %>%
  tibble::column_to_rownames("gene_symbol") %>%
  as.matrix()

Note: If we had duplicate gene symbols here, we couldn’t set them as rownames.

GSVA

Figure 1 from Hänzelmann et al. (2013).

You may notice that GSVA has some commonalities with GSEA. Rather than ranking genes based on some statistic we selected ahead of time, GSVA fits a model and ranks genes based on their expression level relative to the sample distribution. This is a way of asking if a gene i is highly or lowly expressed in a sample j in the context of this experiment and ranking accordingly (Hänzelmann et al. 2013). The pathway-level score calculated is a way of asking how genes within a gene set vary as compared to genes that are outside of that gene set (Malhotra. 2018). (This is sometimes called a competitive test in gene set enrichment literature.) The intuition here is that we will get pathway-level scores for each sample that indicate if genes in a pathway vary concordantly in one direction (overexpressed or underexpressed relative to the overall population) (Hänzelmann et al. 2013).

The output is a gene set by sample matrix of GSVA scores.

Perform GSVA

gsva_results <- gsva(rnaseq_mat, 
                     hallmarks_list,
                     method = "gsva",
                     # Appropriate for our transformed data on log2-like scale
                     kcdf = "Gaussian",  
                     # Minimum gene set size
                     min.sz = 15, 
                     # Maximum gene set size
                     max.sz = 500,
                     # Compute Gaussian-distributed scores
                     mx.diff = TRUE)
945 genes with constant expression values throuhgout the samples.Since argument method!="ssgsea", genes with constant expression values are discarded.
Estimating GSVA scores for 50 gene sets.
Estimating ECDFs with Gaussian kernels

  |                                                                                                               
  |                                                                                                         |   0%
  |                                                                                                               
  |==                                                                                                       |   2%
  |                                                                                                               
  |====                                                                                                     |   4%
  |                                                                                                               
  |======                                                                                                   |   6%
  |                                                                                                               
  |========                                                                                                 |   8%
  |                                                                                                               
  |==========                                                                                               |  10%
  |                                                                                                               
  |=============                                                                                            |  12%
  |                                                                                                               
  |===============                                                                                          |  14%
  |                                                                                                               
  |=================                                                                                        |  16%
  |                                                                                                               
  |===================                                                                                      |  18%
  |                                                                                                               
  |=====================                                                                                    |  20%
  |                                                                                                               
  |=======================                                                                                  |  22%
  |                                                                                                               
  |=========================                                                                                |  24%
  |                                                                                                               
  |===========================                                                                              |  26%
  |                                                                                                               
  |=============================                                                                            |  28%
  |                                                                                                               
  |================================                                                                         |  30%
  |                                                                                                               
  |==================================                                                                       |  32%
  |                                                                                                               
  |====================================                                                                     |  34%
  |                                                                                                               
  |======================================                                                                   |  36%
  |                                                                                                               
  |========================================                                                                 |  38%
  |                                                                                                               
  |==========================================                                                               |  40%
  |                                                                                                               
  |============================================                                                             |  42%
  |                                                                                                               
  |==============================================                                                           |  44%
  |                                                                                                               
  |================================================                                                         |  46%
  |                                                                                                               
  |==================================================                                                       |  48%
  |                                                                                                               
  |====================================================                                                     |  50%
  |                                                                                                               
  |=======================================================                                                  |  52%
  |                                                                                                               
  |=========================================================                                                |  54%
  |                                                                                                               
  |===========================================================                                              |  56%
  |                                                                                                               
  |=============================================================                                            |  58%
  |                                                                                                               
  |===============================================================                                          |  60%
  |                                                                                                               
  |=================================================================                                        |  62%
  |                                                                                                               
  |===================================================================                                      |  64%
  |                                                                                                               
  |=====================================================================                                    |  66%
  |                                                                                                               
  |=======================================================================                                  |  68%
  |                                                                                                               
  |==========================================================================                               |  70%
  |                                                                                                               
  |============================================================================                             |  72%
  |                                                                                                               
  |==============================================================================                           |  74%
  |                                                                                                               
  |================================================================================                         |  76%
  |                                                                                                               
  |==================================================================================                       |  78%
  |                                                                                                               
  |====================================================================================                     |  80%
  |                                                                                                               
  |======================================================================================                   |  82%
  |                                                                                                               
  |========================================================================================                 |  84%
  |                                                                                                               
  |==========================================================================================               |  86%
  |                                                                                                               
  |============================================================================================             |  88%
  |                                                                                                               
  |==============================================================================================           |  90%
  |                                                                                                               
  |=================================================================================================        |  92%
  |                                                                                                               
  |===================================================================================================      |  94%
  |                                                                                                               
  |=====================================================================================================    |  96%
  |                                                                                                               
  |=======================================================================================================  |  98%
  |                                                                                                               
  |=========================================================================================================| 100%

Note: the gsva() documentation says we can use kcdf = "Gaussian" if we had RNA-seq log-CPMs, log-RPKMs or log-TPMs, but we would use kcdf = "Poisson" on integer counts.

# Let's explore what the output of gsva() looks like
gsva_results[1:5, 1:5]
                                    BS_09Z7TC35 BS_1AYRM596 BS_1BWP5MCT BS_1QXEC43H BS_1TWCV047
HALLMARK_TNFA_SIGNALING_VIA_NFKB    -0.44911439   0.5208667  -0.5609193  0.32300630  -0.1468081
HALLMARK_HYPOXIA                    -0.38297104   0.2436910  -0.5058759  0.36247083  -0.2971559
HALLMARK_CHOLESTEROL_HOMEOSTASIS    -0.26534735   0.1054224  -0.5180933  0.32418657  -0.4561386
HALLMARK_MITOTIC_SPINDLE             0.12727006   0.2339489  -0.4338076 -0.07023068  -0.1861483
HALLMARK_WNT_BETA_CATENIN_SIGNALING -0.09287646   0.1602124  -0.4427594  0.41372523  -0.1152437

A note on gene set size

Often the scores of gene set enrichment methods are not comparable between gene sets of different sizes. Let’s do an experiment using randomly generated gene sets to explore this idea a bit more.

We need to get a collection of all possible genes we will sample from to create random gene sets. Because we’re doing some random sampling, we need to set a seed for this to be reproducible.

# Use all the gene symbols in the dataset as the pool of possible genes
all_genes <- rownames(rnaseq_mat)

# Set a seed for reproducibility
set.seed(2020)

Our minimum gene set size earlier was 15 genes and our maximum gene set size was 500 genes. We’ll use the same minimum and maximum values for our random gene sets and some values in between.

# Make a list of integers that indicate the random gene set sizes
gene_set_size <- list(15, 25, 50, 100, 250, 500)

For each gene set size, we will generate 100 random gene sets.

# Set number of replicates
nreps <- 100
# Generate 100 random gene sets of each size
random_gene_sets <- 
  purrr::map(
    rep(gene_set_size, nreps),  # Repeat gene sizes so we run `nreps` times 
    # Sample the vector of all genes, choosing the number of items specified
    # in the element of gene set size
    ~ base::sample(x = all_genes,
                   size = .x)
  )

The Hallmarks list we used earlier stored the gene set names as the name of the list, so let’s add names to our random gene sets that indicate what size they are and so gsva() doesn’t get upset.

# We will include the size of the gene set in the gene set name 
# Start by taking the length of each pathway and appending "pathway_" to that
# number
lengths_vector <- purrr::map(random_gene_sets, ~ length(.x)) %>%
  purrr::map(~ paste0("pathway_", .x)) %>%
  # Return a vector
  purrr::flatten_chr()

# Add the names in lengths_vector to the list
random_gene_sets <- random_gene_sets %>% 
  # make.names() appends a "version" if something is not unique
  purrr::set_names(nm = make.names(lengths_vector, unique = TRUE))

Run GSVA on our dataset with the same parameters as before, but now with random gene sets.

random_gsva_results <- gsva(rnaseq_mat, 
                            random_gene_sets,
                            method = "gsva",
                            # Appropriate for our transformed data on 
                            # log2-like scale
                            kcdf = "Gaussian",  
                            # Minimum gene set size
                            min.sz = 15, 
                            # Maximum gene set size
                            max.sz = 500,
                            # Compute Gaussian-distributed scores
                            mx.diff = TRUE)                            
945 genes with constant expression values throuhgout the samples.Since argument method!="ssgsea", genes with constant expression values are discarded.
Estimating GSVA scores for 579 gene sets.
Estimating ECDFs with Gaussian kernels

  |                                                                                                               
  |                                                                                                         |   0%
  |                                                                                                               
  |=                                                                                                        |   1%
  |                                                                                                               
  |==                                                                                                       |   2%
  |                                                                                                               
  |===                                                                                                      |   2%
  |                                                                                                               
  |===                                                                                                      |   3%
  |                                                                                                               
  |====                                                                                                     |   3%
  |                                                                                                               
  |====                                                                                                     |   4%
  |                                                                                                               
  |=====                                                                                                    |   4%
  |                                                                                                               
  |=====                                                                                                    |   5%
  |                                                                                                               
  |======                                                                                                   |   5%
  |                                                                                                               
  |======                                                                                                   |   6%
  |                                                                                                               
  |=======                                                                                                  |   6%
  |                                                                                                               
  |=======                                                                                                  |   7%
  |                                                                                                               
  |========                                                                                                 |   7%
  |                                                                                                               
  |========                                                                                                 |   8%
  |                                                                                                               
  |=========                                                                                                |   8%
  |                                                                                                               
  |=========                                                                                                |   9%
  |                                                                                                               
  |==========                                                                                               |   9%
  |                                                                                                               
  |==========                                                                                               |  10%
  |                                                                                                               
  |===========                                                                                              |  10%
  |                                                                                                               
  |===========                                                                                              |  11%
  |                                                                                                               
  |============                                                                                             |  11%
  |                                                                                                               
  |============                                                                                             |  12%
  |                                                                                                               
  |=============                                                                                            |  12%
  |                                                                                                               
  |=============                                                                                            |  13%
  |                                                                                                               
  |==============                                                                                           |  13%
  |                                                                                                               
  |==============                                                                                           |  14%
  |                                                                                                               
  |===============                                                                                          |  14%
  |                                                                                                               
  |===============                                                                                          |  15%
  |                                                                                                               
  |================                                                                                         |  15%
  |                                                                                                               
  |================                                                                                         |  16%
  |                                                                                                               
  |=================                                                                                        |  16%
  |                                                                                                               
  |=================                                                                                        |  17%
  |                                                                                                               
  |==================                                                                                       |  17%
  |                                                                                                               
  |==================                                                                                       |  18%
  |                                                                                                               
  |===================                                                                                      |  18%
  |                                                                                                               
  |====================                                                                                     |  19%
  |                                                                                                               
  |====================                                                                                     |  20%
  |                                                                                                               
  |=====================                                                                                    |  20%
  |                                                                                                               
  |======================                                                                                   |  21%
  |                                                                                                               
  |=======================                                                                                  |  22%
  |                                                                                                               
  |========================                                                                                 |  22%
  |                                                                                                               
  |========================                                                                                 |  23%
  |                                                                                                               
  |=========================                                                                                |  23%
  |                                                                                                               
  |=========================                                                                                |  24%
  |                                                                                                               
  |==========================                                                                               |  24%
  |                                                                                                               
  |==========================                                                                               |  25%
  |                                                                                                               
  |===========================                                                                              |  25%
  |                                                                                                               
  |===========================                                                                              |  26%
  |                                                                                                               
  |============================                                                                             |  26%
  |                                                                                                               
  |============================                                                                             |  27%
  |                                                                                                               
  |=============================                                                                            |  27%
  |                                                                                                               
  |=============================                                                                            |  28%
  |                                                                                                               
  |==============================                                                                           |  28%
  |                                                                                                               
  |==============================                                                                           |  29%
  |                                                                                                               
  |===============================                                                                          |  29%
  |                                                                                                               
  |===============================                                                                          |  30%
  |                                                                                                               
  |================================                                                                         |  30%
  |                                                                                                               
  |================================                                                                         |  31%
  |                                                                                                               
  |=================================                                                                        |  31%
  |                                                                                                               
  |=================================                                                                        |  32%
  |                                                                                                               
  |==================================                                                                       |  32%
  |                                                                                                               
  |==================================                                                                       |  33%
  |                                                                                                               
  |===================================                                                                      |  33%
  |                                                                                                               
  |===================================                                                                      |  34%
  |                                                                                                               
  |====================================                                                                     |  34%
  |                                                                                                               
  |====================================                                                                     |  35%
  |                                                                                                               
  |=====================================                                                                    |  35%
  |                                                                                                               
  |=====================================                                                                    |  36%
  |                                                                                                               
  |======================================                                                                   |  36%
  |                                                                                                               
  |======================================                                                                   |  37%
  |                                                                                                               
  |=======================================                                                                  |  37%
  |                                                                                                               
  |========================================                                                                 |  38%
  |                                                                                                               
  |========================================                                                                 |  39%
  |                                                                                                               
  |=========================================                                                                |  39%
  |                                                                                                               
  |==========================================                                                               |  40%
  |                                                                                                               
  |===========================================                                                              |  41%
  |                                                                                                               
  |============================================                                                             |  41%
  |                                                                                                               
  |============================================                                                             |  42%
  |                                                                                                               
  |=============================================                                                            |  42%
  |                                                                                                               
  |=============================================                                                            |  43%
  |                                                                                                               
  |==============================================                                                           |  43%
  |                                                                                                               
  |==============================================                                                           |  44%
  |                                                                                                               
  |===============================================                                                          |  44%
  |                                                                                                               
  |===============================================                                                          |  45%
  |                                                                                                               
  |================================================                                                         |  45%
  |                                                                                                               
  |================================================                                                         |  46%
  |                                                                                                               
  |=================================================                                                        |  46%
  |                                                                                                               
  |=================================================                                                        |  47%
  |                                                                                                               
  |==================================================                                                       |  47%
  |                                                                                                               
  |==================================================                                                       |  48%
  |                                                                                                               
  |===================================================                                                      |  48%
  |                                                                                                               
  |===================================================                                                      |  49%
  |                                                                                                               
  |====================================================                                                     |  49%
  |                                                                                                               
  |====================================================                                                     |  50%
  |                                                                                                               
  |=====================================================                                                    |  50%
  |                                                                                                               
  |=====================================================                                                    |  51%
  |                                                                                                               
  |======================================================                                                   |  51%
  |                                                                                                               
  |======================================================                                                   |  52%
  |                                                                                                               
  |=======================================================                                                  |  52%
  |                                                                                                               
  |=======================================================                                                  |  53%
  |                                                                                                               
  |========================================================                                                 |  53%
  |                                                                                                               
  |========================================================                                                 |  54%
  |                                                                                                               
  |=========================================================                                                |  54%
  |                                                                                                               
  |=========================================================                                                |  55%
  |                                                                                                               
  |==========================================================                                               |  55%
  |                                                                                                               
  |==========================================================                                               |  56%
  |                                                                                                               
  |===========================================================                                              |  56%
  |                                                                                                               
  |===========================================================                                              |  57%
  |                                                                                                               
  |============================================================                                             |  57%
  |                                                                                                               
  |============================================================                                             |  58%
  |                                                                                                               
  |=============================================================                                            |  58%
  |                                                                                                               
  |=============================================================                                            |  59%
  |                                                                                                               
  |==============================================================                                           |  59%
  |                                                                                                               
  |===============================================================                                          |  60%
  |                                                                                                               
  |================================================================                                         |  61%
  |                                                                                                               
  |=================================================================                                        |  61%
  |                                                                                                               
  |=================================================================                                        |  62%
  |                                                                                                               
  |==================================================================                                       |  63%
  |                                                                                                               
  |===================================================================                                      |  63%
  |                                                                                                               
  |===================================================================                                      |  64%
  |                                                                                                               
  |====================================================================                                     |  64%
  |                                                                                                               
  |====================================================================                                     |  65%
  |                                                                                                               
  |=====================================================================                                    |  65%
  |                                                                                                               
  |=====================================================================                                    |  66%
  |                                                                                                               
  |======================================================================                                   |  66%
  |                                                                                                               
  |======================================================================                                   |  67%
  |                                                                                                               
  |=======================================================================                                  |  67%
  |                                                                                                               
  |=======================================================================                                  |  68%
  |                                                                                                               
  |========================================================================                                 |  68%
  |                                                                                                               
  |========================================================================                                 |  69%
  |                                                                                                               
  |=========================================================================                                |  69%
  |                                                                                                               
  |=========================================================================                                |  70%
  |                                                                                                               
  |==========================================================================                               |  70%
  |                                                                                                               
  |==========================================================================                               |  71%
  |                                                                                                               
  |===========================================================================                              |  71%
  |                                                                                                               
  |===========================================================================                              |  72%
  |                                                                                                               
  |============================================================================                             |  72%
  |                                                                                                               
  |============================================================================                             |  73%
  |                                                                                                               
  |=============================================================================                            |  73%
  |                                                                                                               
  |=============================================================================                            |  74%
  |                                                                                                               
  |==============================================================================                           |  74%
  |                                                                                                               
  |==============================================================================                           |  75%
  |                                                                                                               
  |===============================================================================                          |  75%
  |                                                                                                               
  |===============================================================================                          |  76%
  |                                                                                                               
  |================================================================================                         |  76%
  |                                                                                                               
  |================================================================================                         |  77%
  |                                                                                                               
  |=================================================================================                        |  77%
  |                                                                                                               
  |=================================================================================                        |  78%
  |                                                                                                               
  |==================================================================================                       |  78%
  |                                                                                                               
  |===================================================================================                      |  79%
  |                                                                                                               
  |====================================================================================                     |  80%
  |                                                                                                               
  |=====================================================================================                    |  80%
  |                                                                                                               
  |=====================================================================================                    |  81%
  |                                                                                                               
  |======================================================================================                   |  82%
  |                                                                                                               
  |=======================================================================================                  |  82%
  |                                                                                                               
  |=======================================================================================                  |  83%
  |                                                                                                               
  |========================================================================================                 |  83%
  |                                                                                                               
  |========================================================================================                 |  84%
  |                                                                                                               
  |=========================================================================================                |  84%
  |                                                                                                               
  |=========================================================================================                |  85%
  |                                                                                                               
  |==========================================================================================               |  85%
  |                                                                                                               
  |==========================================================================================               |  86%
  |                                                                                                               
  |===========================================================================================              |  86%
  |                                                                                                               
  |===========================================================================================              |  87%
  |                                                                                                               
  |============================================================================================             |  87%
  |                                                                                                               
  |============================================================================================             |  88%
  |                                                                                                               
  |=============================================================================================            |  88%
  |                                                                                                               
  |=============================================================================================            |  89%
  |                                                                                                               
  |==============================================================================================           |  89%
  |                                                                                                               
  |==============================================================================================           |  90%
  |                                                                                                               
  |===============================================================================================          |  90%
  |                                                                                                               
  |===============================================================================================          |  91%
  |                                                                                                               
  |================================================================================================         |  91%
  |                                                                                                               
  |================================================================================================         |  92%
  |                                                                                                               
  |=================================================================================================        |  92%
  |                                                                                                               
  |=================================================================================================        |  93%
  |                                                                                                               
  |==================================================================================================       |  93%
  |                                                                                                               
  |==================================================================================================       |  94%
  |                                                                                                               
  |===================================================================================================      |  94%
  |                                                                                                               
  |===================================================================================================      |  95%
  |                                                                                                               
  |====================================================================================================     |  95%
  |                                                                                                               
  |====================================================================================================     |  96%
  |                                                                                                               
  |=====================================================================================================    |  96%
  |                                                                                                               
  |=====================================================================================================    |  97%
  |                                                                                                               
  |======================================================================================================   |  97%
  |                                                                                                               
  |======================================================================================================   |  98%
  |                                                                                                               
  |=======================================================================================================  |  98%
  |                                                                                                               
  |======================================================================================================== |  99%
  |                                                                                                               
  |=========================================================================================================| 100%

Now let’s make a plot to look at the distribution of scores from random gene sets. First we need to get this data in an appropriate format for ggplot2.

# The random results are a matrix
random_long_df <- random_gsva_results %>%
  data.frame() %>%
  # Gene set names are rownames
  tibble::rownames_to_column("gene_set") %>%
  # Get into long format
  tidyr::pivot_longer(cols = -gene_set, 
                      names_to = "Kids_First_Biospecimen_ID",
                      values_to = "gsva_score") %>%
  # Remove the .version added by make.names()
  dplyr::mutate(gene_set = stringr::str_remove(gene_set, "\\..*")) %>%
  # Add a column that keeps track of the gene set size
  dplyr::mutate(gene_set_size = stringr::word(gene_set, 2, sep = "_")) %>%
  # We want to plot smallest no. genes -> largest no. genes
  dplyr::mutate(gene_set_size = factor(gene_set_size, 
                                       levels = c(15, 25, 50, 100, 250, 500)))

Let’s make a violin plot so we can look at the distribution of scores by gene set size.

# Violin plot comparing GSVA scores of different random gene set sizes
random_long_df %>%
  ggplot2::ggplot(ggplot2::aes(x = gene_set_size, 
                               y = gsva_score)) +
  # Make a violin plot that's a pretty blue!
  ggplot2::geom_violin(fill = "#99CCFF", alpha = 0.5) +
  # Add a point with the mean value
  ggplot2::stat_summary(
    geom = "point",
    fun = "mean",
    # Change the aesthetics of the points
    size = 3,
    color = "#0066CC",
    shape = 18
  ) +
  # Flip the axes
  ggplot2::coord_flip() +
  ggplot2::labs(title = "Random Gene Set GSVA Scores",
                x = "gene set size",
                y = "GSVA score") +
  ggplot2::theme_bw()

What do you notice about these distributions? How might you use this information to inform your interpretation of GSVA scores?

How can you use these scores?

If you did have groups information for your samples, you could test for pathway scores differences between groups. Here’s an example of that from the OpenPBTA project itself!

You can also visualize this matrix in a heatmap. Here’s a figure from the OpenPBTA project, where the middle panel is a heatmap of GSVA scores that were significantly different between histologies.

Code here.

Write results to file

gsva_results %>%
  as.data.frame() %>%
  tibble::rownames_to_column("pathway") %>%
  readr::write_tsv(file = gsva_results_file)

Session Info

sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8    
 [5] LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8       LC_NAME=C             
 [9] LC_ADDRESS=C           LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
[1] GSVA_1.38.2    magrittr_2.0.1

loaded via a namespace (and not attached):
  [1] fgsea_1.16.0                colorspace_2.0-0            ellipsis_0.3.1             
  [4] estimability_1.3            qvalue_2.22.0               XVector_0.30.0             
  [7] GenomicRanges_1.42.0        rstudioapi_0.13             farver_2.0.3               
 [10] graphlayouts_0.7.1          ggrepel_0.9.1               bit64_4.0.5                
 [13] fansi_0.4.2                 AnnotationDbi_1.52.0        mvtnorm_1.1-1              
 [16] scatterpie_0.1.5            splines_4.0.3               cachem_1.0.1               
 [19] GOSemSim_2.16.1             knitr_1.30                  polyclip_1.10-0            
 [22] jsonlite_1.7.2              annotate_1.68.0             GO.db_3.12.1               
 [25] graph_1.68.0                ggforce_0.3.2               BiocManager_1.30.10        
 [28] readr_1.4.0                 compiler_4.0.3              httr_1.4.2                 
 [31] rvcheck_0.1.8               emmeans_1.5.3               assertthat_0.2.1           
 [34] Matrix_1.3-2                fastmap_1.1.0               cli_2.2.0                  
 [37] limma_3.46.0                tweenr_1.0.1                htmltools_0.5.1.1          
 [40] tools_4.0.3                 igraph_1.2.6                coda_0.19-4                
 [43] gtable_0.3.0                glue_1.4.2                  GenomeInfoDbData_1.2.4     
 [46] reshape2_1.4.4              DO.db_2.9                   dplyr_1.0.3                
 [49] fastmatch_1.1-0             Rcpp_1.0.6                  enrichplot_1.10.2          
 [52] Biobase_2.50.0              vctrs_0.3.6                 nlme_3.1-151               
 [55] ggraph_2.0.4                xfun_0.20                   stringr_1.4.0              
 [58] qusage_2.24.0               lifecycle_0.2.0             renv_0.12.5-2              
 [61] XML_3.99-0.5                DOSE_3.16.0                 zlibbioc_1.36.0            
 [64] MASS_7.3-53                 scales_1.1.1                tidygraph_1.2.0            
 [67] hms_1.0.0                   MatrixGenerics_1.2.0        parallel_4.0.3             
 [70] SummarizedExperiment_1.20.0 RColorBrewer_1.1-2          yaml_2.2.1                 
 [73] memoise_1.1.0               gridExtra_2.3               ggplot2_3.3.3              
 [76] stringi_1.5.3               RSQLite_2.2.3               S4Vectors_0.28.1           
 [79] BiocGenerics_0.36.0         BiocParallel_1.24.1         GenomeInfoDb_1.26.2        
 [82] rlang_0.4.10                pkgconfig_2.0.3             matrixStats_0.57.0         
 [85] bitops_1.0-6                evaluate_0.14               lattice_0.20-41            
 [88] purrr_0.3.4                 labeling_0.4.2              cowplot_1.1.1              
 [91] shadowtext_0.0.7            bit_4.0.4                   tidyselect_1.1.0           
 [94] GSEABase_1.52.1             plyr_1.8.6                  R6_2.5.0                   
 [97] IRanges_2.24.1              fftw_1.0-6                  generics_0.1.0             
[100] DelayedArray_0.16.2         DBI_1.1.1                   pillar_1.4.7               
[103] RCurl_1.98-1.2              tibble_3.0.5                crayon_1.3.4               
[106] rmarkdown_2.6               viridis_0.5.1               grid_4.0.3                 
[109] data.table_1.13.6           blob_1.2.1                  digest_0.6.27              
[112] xtable_1.8-4                tidyr_1.1.2                 stats4_4.0.3               
[115] munsell_0.5.0               viridisLite_0.3.0          
LS0tCnRpdGxlOiAiUGF0aHdheSBhbmFseXNpczogR2VuZSBTZXQgVmFyaWF0aW9uIEFuYWx5c2lzIChHU1ZBKSIKb3V0cHV0OiAKICBodG1sX25vdGVib29rOgogICAgdG9jOiB0cnVlCiAgICB0b2NfZmxvYXQ6IHRydWUKYXV0aG9yOiBDQ0RMIGZvciBBTFNGCmRhdGU6IDIwMjAKLS0tCgojIyBPYmplY3RpdmVzCgpUaGlzIG5vdGVib29rIHdpbGwgZGVtb25zdHJhdGUgaG93IHRvOgoKLSBJZGVudGlmeSB3aGVuIEdlbmUgU2V0IFZhcmlhdGlvbiBBbmFseXNpcyAoR1NWQSkgaXMgd2VsbC1zdWl0ZWQgZm9yIGFuIGFuYWx5c2lzCi0gUGVyZm9ybSBHU1ZBIG9uIHRyYW5zZm9ybWVkIFJOQS1zZXEgZGF0YSB3aXRoIHRoZSBgR1NWQWAgcGFja2FnZQotIEdlbmVyYXRlIHJhbmRvbSBnZW5lIHNldHMgdXNpbmcgZnVuY3Rpb25hbCBwcm9ncmFtbWluZyB0b29scyBmcm9tIHRoZSBbYHB1cnJyYCBwYWNrYWdlXShodHRwczovL3B1cnJyLnRpZHl2ZXJzZS5vcmcvKQoKLS0tCgpTbyBmYXIgZXZlcnkgcGF0aHdheSBhbmFseXNpcyBtZXRob2Qgd2UndmUgY292ZXJlZCByZWxpZXMgb24gc29tZSBpbmZvcm1hdGlvbiBhYm91dCBncm91cHMgb2Ygc2FtcGxlcyBpbiBvdXIgZGF0YS4KRm9yIG92ZXItcmVwcmVzZW50YXRpb24gYW5hbHlzaXMgKE9SQSksIHdlIGNyZWF0ZWQgZ2VuZSBzZXRzIGZyb20gdHdvIGRpZmZlcmVudCB0d28gZ3JvdXAgY29tcGFyaXNvbnMuCkluIHRoZSBHZW5lIFNldCBFbnJpY2htZW50IEFuYWx5c2lzIChHU0VBKSBleGFtcGxlLCB3ZSB1c2VkIHN0YXRpc3RpY3MgZnJvbSBhIGRpZmZlcmVudGlhbCBnZW5lIGV4cHJlc3Npb24gKERHRSkgYW5hbHlzaXMgd2hlcmUgd2UgY29tcGFyZWQgX01ZQ05fIGFtcGxpZmllZCBjZWxsIGxpbmVzIHRvIG5vbi1hbXBsaWZpZWQgY2VsbCBsaW5lczsgd2UgbmVlZGVkIHRoYXQgYW1wbGlmaWNhdGlvbiBzdGF0dXMgaW5mb3JtYXRpb24uCgpXaGF0IGlmIHdlJ3JlIGxlc3Mgc3VyZSBhYm91dCBncm91cHMgaW4gb3VyIGRhdGEgb3Igd2Ugd2FudCB0byBhbmFseXplIG91ciBkYXRhIGluIGEgbW9yZSB1bnN1cGVydmlzZWQgbWFubmVyPwoKSW4gdGhpcyBub3RlYm9vayB3ZSB3aWxsIGNvdmVyIGEgbWV0aG9kIGNhbGxlZCBHZW5lIFNldCBWYXJpYXRpb24gQW5hbHlzaXMgKEdTVkEpIChbSMOkbnplbG1hbm4gX2V0IGFsLl8gMjAxM10oaHR0cHM6Ly9kb2kub3JnLzEwLjExODYvMTQ3MS0yMTA1LTE0LTcpKSB0aGF0IGFsbG93cyB1cyB0byBjYWxjdWxhdGUgZ2VuZSBzZXQgb3IgcGF0aHdheSBzY29yZXMgb24gYSBwZXItc2FtcGxlIGJhc2lzLgoKV2UgbGlrZSB0aGlzIHF1b3RlIGZyb20gdGhlIEdTVkEgcGFwZXIgKFtIw6RuemVsbWFubiBfZXQgYWwuXyAyMDEzXShodHRwczovL2RvaS5vcmcvMTAuMTE4Ni8xNDcxLTIxMDUtMTQtNykpIHRvIHNldCB0aGUgc3RhZ2U6Cgo+IFdoaWxlIFtnZW5lIHNldCBlbnJpY2htZW50XSBtZXRob2RzIGFyZSBnZW5lcmFsbHkgcmVnYXJkZWQgYXMgZW5kIHBvaW50cyBvZiBhIGJpb2luZm9ybWF0aWMgYW5hbHlzaXMsIEdTVkEgY29uc3RpdHV0ZXMgYSBzdGFydGluZyBwb2ludCB0byBidWlsZCBwYXRod2F5LWNlbnRyaWMgbW9kZWxzIG9mIGJpb2xvZ3kuCgpSYXRoZXIgdGhhbiBjb250ZXh0dWFsaXppbmcgc29tZSByZXN1bHRzIHlvdSBfYWxyZWFkeSBoYXZlXyBmcm9tIGFub3RoZXIgYW5hbHlzaXMgbGlrZSBER0UsIEdTVkEgaXMgZGVzaWduZWQgdG8gcHJvdmlkZSBhbiBlc3RpbWF0ZSBvZiBwYXRod2F5IHZhcmlhdGlvbiBmb3IgZWFjaCBvZiB0aGUgc2FtcGxlcyBpbiBhbiBleHBlcmltZW50LgpOb3RlIHRoYXQgdGhlc2Ugc2NvcmVzIHdpbGwgZGVwZW5kIG9uIHRoZSBzYW1wbGVzIGluY2x1ZGVkIGluIHRoZSBkYXRhc2V0IHdoZW4geW91IHJ1biBHU1ZBOyBpZiB5b3UgYWRkZWQgbW9yZSBzYW1wbGVzIGFuZCByZXJhbiBHU1ZBLCB5b3Ugd291bGQgZXhwZWN0IHRoZSBzY29yZXMgdG8gY2hhbmdlLgoKIyMjIyBPdGhlciByZXNvdXJjZXMKCiogW01hbGhvdHJhLiBfRGVjb2RpbmcgR2VuZSBTZXQgVmFyaWF0aW9uIEFuYWx5c2lzXy4gKDIwMTgpXShodHRwczovL3Rvd2FyZHNkYXRhc2NpZW5jZS5jb20vZGVjb2RpbmctZ2VuZS1zZXQtdmFyaWF0aW9uLWFuYWx5c2lzLTgxOTNhMGNmZGEzKQoKIyMgU2V0IHVwCgojIyMgTGlicmFyaWVzCgpgYGB7ciBsaWJyYXJpZXN9CiMgUGlwZXMKbGlicmFyeShtYWdyaXR0cikKIyBHZW5lIFNldCBWYXJpYXRpb24gQW5hbHlzaXMKbGlicmFyeShHU1ZBKQpgYGAKCiMjIyBEaXJlY3RvcmllcyBhbmQgZmlsZXMKCiMjIyMgRGlyZWN0b3JpZXMKCmBgYHtyIGRpcmVjdG9yaWVzfQojIFdlIGhhdmUgc29tZSBtZWR1bGxvYmxhc3RvbWEgZGF0YSBmcm9tIHRoZSBPcGVuUEJUQSBwcm9qZWN0IHRoYXQgd2UndmUgCiMgcHJlcGFyZWQgYWhlYWQgb2YgdGltZQppbnB1dF9kaXIgPC0gZmlsZS5wYXRoKCJkYXRhIiwgIm9wZW4tcGJ0YSIpCgojIENyZWF0ZSBhIGRpcmVjdG9yeSBzcGVjaWZpY2FsbHkgZm9yIHRoZSByZXN1bHRzIHVzaW5nIHRoaXMgZGF0YXNldApvdXRwdXRfZGlyIDwtIGZpbGUucGF0aCgicmVzdWx0cyIsICJvcGVuLXBidGEiKSAKaWYgKCFkaXIuZXhpc3RzKG91dHB1dF9kaXIpKSB7CiAgZGlyLmNyZWF0ZShvdXRwdXRfZGlyLCByZWN1cnNpdmUgPSBUUlVFKQp9CmBgYAoKIyMjIyBJbnB1dAoKV2UgaGF2ZSBWU1QgdHJhbnNmb3JtZWQgUk5BLXNlcSBkYXRhLCBhbm5vdGF0ZWQgd2l0aCBnZW5lIHN5bWJvbHMsIHRoYXQgaGFzIGJlZW4gY29sbGFwc2VkIHN1Y2ggdGhhdCB0aGVyZSBhcmUgbm8gZHVwbGljYXRlZCBnZW5lIGlkZW50aWZpZXJzIChzZWUgYHNldHVwYCkuCgpgYGB7ciBpbnB1dF9maWxlfQpybmFzZXFfZmlsZSA8LSBmaWxlLnBhdGgoaW5wdXRfZGlyLCAibWVkdWxsb2JsYXN0b21hX3ZzdF9jb2xsYXBzZWQudHN2IikKYGBgCgojIyMjIE91dHB1dAoKYGBge3Igb3V0cHV0X2ZpbGUsIGxpdmUgPSBUUlVFfQpnc3ZhX3Jlc3VsdHNfZmlsZSA8LSBmaWxlLnBhdGgob3V0cHV0X2RpciwgIm1lZHVsbG9ibGFzdG9tYV9nc3ZhX3Jlc3VsdHMudHN2IikKYGBgCgojIyBHZW5lIHNldHMKClRoZSBmdW5jdGlvbiB0aGF0IHdlIHdpbGwgdXNlIHRvIHJ1biBHU1ZBIHdhbnRzIHRoZSBnZW5lIHNldHMgdG8gYmUgaW4gYSBsaXN0LCByYXRoZXIgdGhhbiBhIHRpZHkgZGF0YSBmcmFtZSB0aGF0IHdlIHVzZWQgd2l0aCBgY2x1c3RlclByb2ZpbGVyYCAoYWx0aG91Z2ggaXQgZG9lcyBhY2NlcHQgb3RoZXIgZm9ybWF0cykuCgpXZSdyZSBnb2luZyB0byB0YWtlIHRoaXMgb3Bwb3J0dW5pdHkgdG8gaW50cm9kdWNlIGEgZGlmZmVyZW50IGZvcm1hdCB0aGF0IGdlbmUgc2V0cyBhcmUgb2Z0ZW4gZGlzdHJpYnV0ZWQgaW4gY2FsbGVkIFtHTVQgKEdlbmUgTWF0cml4IFRyYW5zcG9zZWQpXShodHRwczovL3NvZnR3YXJlLmJyb2FkaW5zdGl0dXRlLm9yZy9jYW5jZXIvc29mdHdhcmUvZ3NlYS93aWtpL2luZGV4LnBocC9EYXRhX2Zvcm1hdHMjR01UOl9HZW5lX01hdHJpeF9UcmFuc3Bvc2VkX2ZpbGVfZm9ybWF0Xy4yOC4yQS5nbXQuMjkpLgoKV2UncmUgZ29pbmcgdG8gcmVhZCBpbiB0aGUgSGFsbG1hcmsgY29sbGVjdGlvbiBmaWxlIGRpcmVjdGx5IGZyb20gW01TaWdEQl0oaHR0cHM6Ly93d3cuZ3NlYS1tc2lnZGIub3JnL2dzZWEvbXNpZ2RiL2luZGV4LmpzcCksIHJhdGhlciB0aGFuIHVzaW5nIGBtc2lnZGJyYCBsaWtlIHdlIGRpZCBpbiBlYXJsaWVyIG5vdGVib29rcy4KClRoZSBSTkEtc2VxIGRhdGEgdXNlcyBnZW5lIHN5bWJvbHMsIHNvIHdlIG5lZWQgZ2VuZSBzZXRzIHRoYXQgdXNlIGdlbmUgc3ltYm9scywgdG9vLgoKYGBge3IgZ210fQojIFIgY2FuIG9mdGVuIHJlYWQgaW4gZGF0YSBmcm9tIGEgVVJMCmhhbGxtYXJrc191cmwgPC0gImh0dHBzOi8vZGF0YS5icm9hZGluc3RpdHV0ZS5vcmcvZ3NlYS1tc2lnZGIvbXNpZ2RiL3JlbGVhc2UvNy4xL2guYWxsLnY3LjEuc3ltYm9scy5nbXQiCgojIFF1U0FHRSBpcyBhbm90aGVyIHBhdGh3YXkgYW5hbHlzaXMgbWV0aG9kLCB0aGUgcXVzYWdlIHBhY2thZ2UgaGFzIGEgZnVuY3Rpb24KIyBmb3IgcmVhZGluZyBHTVQgZmlsZXMgYW5kIHR1cm5pbmcgdGhlbSBpbnRvIGEgbGlzdApoYWxsbWFya3NfbGlzdCA8LSBxdXNhZ2U6OnJlYWQuZ210KGhhbGxtYXJrc191cmwpCmBgYAoKV2hhdCBkb2VzIHRoaXMgbGlzdCBsb29rIGxpa2U/CgpgYGB7ciBoZWFkX2hhbGxtYXJrLCBldmFsID0gRkFMU0V9CmhlYWQoaGFsbG1hcmtzX2xpc3QpCmBgYAoKIyMgUk5BLXNlcSBkYXRhCgpXZSBoYXZlIFZTVCB0cmFuc2Zvcm1lZCBSTkEtc2VxIGRhdGEsIHdoaWNoIGlzIG9uIGEgbG9nMi1saWtlIHNjYWxlLgpUaGVzZSBkYXRhIGFyZSBmcm9tIHRoZSBPcGVuIFBlZGlhdHJpYyBCcmFpbiBUdW1vciBBdGxhcyAoT3BlblBCVEEpCk9wZW5QQlRBIGlzIGEgY29sbGFib3JhdGl2ZSBwcm9qZWN0IG9yZ2FuaXplZCBieSB0aGUgQ0NETCBhbmQgdGhlIENlbnRlciBmb3IgRGF0YS1Ecml2ZW4gRGlzY292ZXJ5IGluIEJpb21lZGljaW5lIChEM2IpIGF0IHRoZSBDaGlsZHJlbidzIEhvc3BpdGFsIG9mIFBoaWxhZGVscGhpYSBjb25kdWN0ZWQgb3Blbmx5IG9uIEdpdEh1Yi4KCllvdSBjYW4gcmVhZCBtb3JlIGFib3V0IHRoZSBwcm9qZWN0IFtoZXJlXShodHRwczovL2dpdGh1Yi5jb20vYWxleHNsZW1vbmFkZS9vcGVucGJ0YS1hbmFseXNpcy8jb3BlbnBidGEtYW5hbHlzaXMpLgoKV2UncmUgb25seSB3b3JraW5nIHdpdGggdGhlIG1lZHVsbG9ibGFzdG9tYSBzYW1wbGVzIGluIHRoaXMgZXhhbXBsZS4KCmBgYHtyIHJlYWRfaW5fcm5hc2VxfQpybmFzZXFfZGYgPC0gcmVhZHI6OnJlYWRfdHN2KHJuYXNlcV9maWxlKQpgYGAKCmBgYHtyIHBlZWtfcm5hc2VxLCBsaXZlID0gVFJVRX0KIyBXaGF0IGRvZXMgdGhlIFJOQS1zZXEgZGF0YSBmcmFtZSBsb29rIGxpa2U/CnJuYXNlcV9kZlsxOjUsIDE6NV0KYGBgCgpGb3IgR1NWQSwgd2UgbmVlZCBhIG1hdHJpeC4KCmBgYHtyIHJuYXNlcV9tYXQsIGxpdmUgPSBUUlVFfQpybmFzZXFfbWF0IDwtIHJuYXNlcV9kZiAlPiUKICB0aWJibGU6OmNvbHVtbl90b19yb3duYW1lcygiZ2VuZV9zeW1ib2wiKSAlPiUKICBhcy5tYXRyaXgoKQpgYGAKCipOb3RlOiBJZiB3ZSBoYWQgZHVwbGljYXRlIGdlbmUgc3ltYm9scyBoZXJlLCB3ZSBjb3VsZG4ndCBzZXQgdGhlbSBhcyByb3duYW1lcy4qCgojIyBHU1ZBCgohW10oZGlhZ3JhbXMvaGFuemVsbWFubl9maWcxLmpwZykKCioqRmlndXJlIDEgZnJvbSBbSMOkbnplbG1hbm4gX2V0IGFsLl8gKDIwMTMpXShodHRwczovL2RvaS5vcmcvMTAuMTE4Ni8xNDcxLTIxMDUtMTQtNykuKioKCllvdSBtYXkgbm90aWNlIHRoYXQgR1NWQSBoYXMgc29tZSBjb21tb25hbGl0aWVzIHdpdGggR1NFQS4KUmF0aGVyIHRoYW4gcmFua2luZyBnZW5lcyBiYXNlZCBvbiBzb21lIHN0YXRpc3RpYyBfd2VfIHNlbGVjdGVkIGFoZWFkIG9mIHRpbWUsIEdTVkEgZml0cyBhIG1vZGVsIGFuZCByYW5rcyBnZW5lcyBiYXNlZCBvbiB0aGVpciBleHByZXNzaW9uIGxldmVsIHJlbGF0aXZlIHRvIHRoZSBzYW1wbGUgZGlzdHJpYnV0aW9uLgpUaGlzIGlzIGEgd2F5IG9mIGFza2luZyBpZiBhIGdlbmUgX2lfIGlzIGhpZ2hseSBvciBsb3dseSBleHByZXNzZWQgaW4gYSBzYW1wbGUgX2pfIGluIHRoZSBjb250ZXh0IG9mIHRoaXMgZXhwZXJpbWVudCBhbmQgcmFua2luZyBhY2NvcmRpbmdseSAoW0jDpG56ZWxtYW5uIF9ldCBhbC5fIDIwMTNdKGh0dHBzOi8vZG9pLm9yZy8xMC4xMTg2LzE0NzEtMjEwNS0xNC03KSkuClRoZSBwYXRod2F5LWxldmVsIHNjb3JlIGNhbGN1bGF0ZWQgaXMgYSB3YXkgb2YgYXNraW5nIGhvdyBnZW5lcyBfd2l0aGluXyBhIGdlbmUgc2V0IHZhcnkgYXMgY29tcGFyZWQgdG8gZ2VuZXMgdGhhdCBhcmUgX291dHNpZGVfIG9mIHRoYXQgZ2VuZSBzZXQgKFtNYWxob3RyYS4gMjAxOF0oaHR0cHM6Ly90b3dhcmRzZGF0YXNjaWVuY2UuY29tL2RlY29kaW5nLWdlbmUtc2V0LXZhcmlhdGlvbi1hbmFseXNpcy04MTkzYTBjZmRhMykpLiAKKFRoaXMgaXMgc29tZXRpbWVzIGNhbGxlZCBhIGNvbXBldGl0aXZlIHRlc3QgaW4gZ2VuZSBzZXQgZW5yaWNobWVudCBsaXRlcmF0dXJlLikKVGhlIGludHVpdGlvbiBoZXJlIGlzIHRoYXQgd2Ugd2lsbCBnZXQgcGF0aHdheS1sZXZlbCBzY29yZXMgZm9yIGVhY2ggc2FtcGxlIHRoYXQgaW5kaWNhdGUgaWYgZ2VuZXMgaW4gYSBwYXRod2F5IHZhcnkgY29uY29yZGFudGx5IGluIG9uZSBkaXJlY3Rpb24gKG92ZXJleHByZXNzZWQgb3IgdW5kZXJleHByZXNzZWQgcmVsYXRpdmUgdG8gdGhlIG92ZXJhbGwgcG9wdWxhdGlvbikgKFtIw6RuemVsbWFubiBfZXQgYWwuXyAyMDEzXShodHRwczovL2RvaS5vcmcvMTAuMTE4Ni8xNDcxLTIxMDUtMTQtNykpLgoKVGhlIG91dHB1dCBpcyBhIGdlbmUgc2V0IGJ5IHNhbXBsZSBtYXRyaXggb2YgR1NWQSBzY29yZXMuCgojIyMgUGVyZm9ybSBHU1ZBCgpgYGB7ciBydW5fZ3N2YX0KZ3N2YV9yZXN1bHRzIDwtIGdzdmEocm5hc2VxX21hdCwgCiAgICAgICAgICAgICAgICAgICAgIGhhbGxtYXJrc19saXN0LAogICAgICAgICAgICAgICAgICAgICBtZXRob2QgPSAiZ3N2YSIsCiAgICAgICAgICAgICAgICAgICAgICMgQXBwcm9wcmlhdGUgZm9yIG91ciB0cmFuc2Zvcm1lZCBkYXRhIG9uIGxvZzItbGlrZSBzY2FsZQogICAgICAgICAgICAgICAgICAgICBrY2RmID0gIkdhdXNzaWFuIiwgIAogICAgICAgICAgICAgICAgICAgICAjIE1pbmltdW0gZ2VuZSBzZXQgc2l6ZQogICAgICAgICAgICAgICAgICAgICBtaW4uc3ogPSAxNSwgCiAgICAgICAgICAgICAgICAgICAgICMgTWF4aW11bSBnZW5lIHNldCBzaXplCiAgICAgICAgICAgICAgICAgICAgIG1heC5zeiA9IDUwMCwKICAgICAgICAgICAgICAgICAgICAgIyBDb21wdXRlIEdhdXNzaWFuLWRpc3RyaWJ1dGVkIHNjb3JlcwogICAgICAgICAgICAgICAgICAgICBteC5kaWZmID0gVFJVRSkKYGBgCgoqKk5vdGU6IHRoZSBgZ3N2YSgpYCBkb2N1bWVudGF0aW9uIHNheXMgd2UgY2FuIHVzZSBga2NkZiA9ICJHYXVzc2lhbiJgIGlmIHdlIGhhZCBSTkEtc2VxIGxvZy1DUE1zLCBsb2ctUlBLTXMgb3IgbG9nLVRQTXMsIGJ1dCB3ZSB3b3VsZCB1c2UgYGtjZGYgPSAiUG9pc3NvbiJgIG9uIGludGVnZXIgY291bnRzLioqCgpgYGB7ciBnc3ZhX3BlZWt9CiMgTGV0J3MgZXhwbG9yZSB3aGF0IHRoZSBvdXRwdXQgb2YgZ3N2YSgpIGxvb2tzIGxpa2UKZ3N2YV9yZXN1bHRzWzE6NSwgMTo1XQpgYGAKCiMjIyBBIG5vdGUgb24gZ2VuZSBzZXQgc2l6ZQogCk9mdGVuIHRoZSBzY29yZXMgb2YgZ2VuZSBzZXQgZW5yaWNobWVudCBtZXRob2RzIGFyZSBub3QgY29tcGFyYWJsZSBiZXR3ZWVuIGdlbmUgc2V0cyBvZiBkaWZmZXJlbnQgc2l6ZXMuCkxldCdzIGRvIGFuIGV4cGVyaW1lbnQgdXNpbmcgcmFuZG9tbHkgZ2VuZXJhdGVkIGdlbmUgc2V0cyB0byBleHBsb3JlIHRoaXMgaWRlYSBhIGJpdCBtb3JlLgoKV2UgbmVlZCB0byBnZXQgYSBjb2xsZWN0aW9uIG9mIGFsbCBwb3NzaWJsZSBnZW5lcyB3ZSB3aWxsIHNhbXBsZSBmcm9tIHRvIGNyZWF0ZSByYW5kb20gZ2VuZSBzZXRzLgpCZWNhdXNlIHdlJ3JlIGRvaW5nIHNvbWUgcmFuZG9tIHNhbXBsaW5nLCB3ZSBuZWVkIHRvIHNldCBhIHNlZWQgZm9yIHRoaXMgdG8gYmUgcmVwcm9kdWNpYmxlLgoKYGBge3IgcmFuZG9tX3NldHVwfQojIFVzZSBhbGwgdGhlIGdlbmUgc3ltYm9scyBpbiB0aGUgZGF0YXNldCBhcyB0aGUgcG9vbCBvZiBwb3NzaWJsZSBnZW5lcwphbGxfZ2VuZXMgPC0gcm93bmFtZXMocm5hc2VxX21hdCkKCiMgU2V0IGEgc2VlZCBmb3IgcmVwcm9kdWNpYmlsaXR5CnNldC5zZWVkKDIwMjApCmBgYAoKT3VyIG1pbmltdW0gZ2VuZSBzZXQgc2l6ZSBlYXJsaWVyIHdhcyAxNSBnZW5lcyBhbmQgb3VyIG1heGltdW0gZ2VuZSBzZXQgc2l6ZSB3YXMgNTAwIGdlbmVzLiAKV2UnbGwgdXNlIHRoZSBzYW1lIG1pbmltdW0gYW5kIG1heGltdW0gdmFsdWVzIGZvciBvdXIgcmFuZG9tIGdlbmUgc2V0cyBhbmQgc29tZSB2YWx1ZXMgaW4gYmV0d2Vlbi4KCmBgYHtyIGdlbmVfc2V0X3NpemVzfQojIE1ha2UgYSBsaXN0IG9mIGludGVnZXJzIHRoYXQgaW5kaWNhdGUgdGhlIHJhbmRvbSBnZW5lIHNldCBzaXplcwpnZW5lX3NldF9zaXplIDwtIGxpc3QoMTUsIDI1LCA1MCwgMTAwLCAyNTAsIDUwMCkKYGBgCgpGb3IgZWFjaCBnZW5lIHNldCBzaXplLCB3ZSB3aWxsIGdlbmVyYXRlIDEwMCByYW5kb20gZ2VuZSBzZXRzLgoKYGBge3IgcmFuZG9tX2dlbmVfc2V0c30KIyBTZXQgbnVtYmVyIG9mIHJlcGxpY2F0ZXMKbnJlcHMgPC0gMTAwCiMgR2VuZXJhdGUgMTAwIHJhbmRvbSBnZW5lIHNldHMgb2YgZWFjaCBzaXplCnJhbmRvbV9nZW5lX3NldHMgPC0gCiAgcHVycnI6Om1hcCgKICAgIHJlcChnZW5lX3NldF9zaXplLCBucmVwcyksICAjIFJlcGVhdCBnZW5lIHNpemVzIHNvIHdlIHJ1biBgbnJlcHNgIHRpbWVzIAogICAgIyBTYW1wbGUgdGhlIHZlY3RvciBvZiBhbGwgZ2VuZXMsIGNob29zaW5nIHRoZSBudW1iZXIgb2YgaXRlbXMgc3BlY2lmaWVkCiAgICAjIGluIHRoZSBlbGVtZW50IG9mIGdlbmUgc2V0IHNpemUKICAgIH4gYmFzZTo6c2FtcGxlKHggPSBhbGxfZ2VuZXMsCiAgICAgICAgICAgICAgICAgICBzaXplID0gLngpCiAgKQpgYGAKClRoZSBIYWxsbWFya3MgbGlzdCB3ZSB1c2VkIGVhcmxpZXIgc3RvcmVkIHRoZSBnZW5lIHNldCBuYW1lcyBhcyB0aGUgbmFtZSBvZiB0aGUgbGlzdCwgc28gbGV0J3MgYWRkIG5hbWVzIHRvIG91ciByYW5kb20gZ2VuZSBzZXRzIHRoYXQgaW5kaWNhdGUgd2hhdCBzaXplIHRoZXkgYXJlIGFuZCBzbyBgZ3N2YSgpYCBkb2Vzbid0IGdldCB1cHNldC4KCmBgYHtyIG5hbWVfcmFuZG9tX2dlbmVfc2V0c30KIyBXZSB3aWxsIGluY2x1ZGUgdGhlIHNpemUgb2YgdGhlIGdlbmUgc2V0IGluIHRoZSBnZW5lIHNldCBuYW1lIAojIFN0YXJ0IGJ5IHRha2luZyB0aGUgbGVuZ3RoIG9mIGVhY2ggcGF0aHdheSBhbmQgYXBwZW5kaW5nICJwYXRod2F5XyIgdG8gdGhhdAojIG51bWJlcgpsZW5ndGhzX3ZlY3RvciA8LSBwdXJycjo6bWFwKHJhbmRvbV9nZW5lX3NldHMsIH4gbGVuZ3RoKC54KSkgJT4lCiAgcHVycnI6Om1hcCh+IHBhc3RlMCgicGF0aHdheV8iLCAueCkpICU+JQogICMgUmV0dXJuIGEgdmVjdG9yCiAgcHVycnI6OmZsYXR0ZW5fY2hyKCkKCiMgQWRkIHRoZSBuYW1lcyBpbiBsZW5ndGhzX3ZlY3RvciB0byB0aGUgbGlzdApyYW5kb21fZ2VuZV9zZXRzIDwtIHJhbmRvbV9nZW5lX3NldHMgJT4lIAogICMgbWFrZS5uYW1lcygpIGFwcGVuZHMgYSAidmVyc2lvbiIgaWYgc29tZXRoaW5nIGlzIG5vdCB1bmlxdWUKICBwdXJycjo6c2V0X25hbWVzKG5tID0gbWFrZS5uYW1lcyhsZW5ndGhzX3ZlY3RvciwgdW5pcXVlID0gVFJVRSkpCmBgYAogClJ1biBHU1ZBIG9uIG91ciBkYXRhc2V0IHdpdGggdGhlIHNhbWUgcGFyYW1ldGVycyBhcyBiZWZvcmUsIGJ1dCBub3cgd2l0aCByYW5kb20gZ2VuZSBzZXRzLgoKYGBge3IgcmFuZG9tX2dzdmEsIGxpdmUgPSBUUlVFfQpyYW5kb21fZ3N2YV9yZXN1bHRzIDwtIGdzdmEocm5hc2VxX21hdCwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICByYW5kb21fZ2VuZV9zZXRzLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgbWV0aG9kID0gImdzdmEiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIyBBcHByb3ByaWF0ZSBmb3Igb3VyIHRyYW5zZm9ybWVkIGRhdGEgb24gCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAjIGxvZzItbGlrZSBzY2FsZQogICAgICAgICAgICAgICAgICAgICAgICAgICAga2NkZiA9ICJHYXVzc2lhbiIsICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgTWluaW11bSBnZW5lIHNldCBzaXplCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBtaW4uc3ogPSAxNSwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAjIE1heGltdW0gZ2VuZSBzZXQgc2l6ZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgbWF4LnN6ID0gNTAwLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIyBDb21wdXRlIEdhdXNzaWFuLWRpc3RyaWJ1dGVkIHNjb3JlcwogICAgICAgICAgICAgICAgICAgICAgICAgICAgbXguZGlmZiA9IFRSVUUpICAgICAgICAgICAgICAgICAgICAgICAgICAgIApgYGAKCk5vdyBsZXQncyBtYWtlIGEgcGxvdCB0byBsb29rIGF0IHRoZSBkaXN0cmlidXRpb24gb2Ygc2NvcmVzIGZyb20gcmFuZG9tIGdlbmUgc2V0cy4KRmlyc3Qgd2UgbmVlZCB0byBnZXQgdGhpcyBkYXRhIGluIGFuIGFwcHJvcHJpYXRlIGZvcm1hdCBmb3IgYGdncGxvdDJgLgoKYGBge3IgbG9uZ2VyX3JhbmRvbV9nc3ZhfQojIFRoZSByYW5kb20gcmVzdWx0cyBhcmUgYSBtYXRyaXgKcmFuZG9tX2xvbmdfZGYgPC0gcmFuZG9tX2dzdmFfcmVzdWx0cyAlPiUKICBkYXRhLmZyYW1lKCkgJT4lCiAgIyBHZW5lIHNldCBuYW1lcyBhcmUgcm93bmFtZXMKICB0aWJibGU6OnJvd25hbWVzX3RvX2NvbHVtbigiZ2VuZV9zZXQiKSAlPiUKICAjIEdldCBpbnRvIGxvbmcgZm9ybWF0CiAgdGlkeXI6OnBpdm90X2xvbmdlcihjb2xzID0gLWdlbmVfc2V0LCAKICAgICAgICAgICAgICAgICAgICAgIG5hbWVzX3RvID0gIktpZHNfRmlyc3RfQmlvc3BlY2ltZW5fSUQiLAogICAgICAgICAgICAgICAgICAgICAgdmFsdWVzX3RvID0gImdzdmFfc2NvcmUiKSAlPiUKICAjIFJlbW92ZSB0aGUgLnZlcnNpb24gYWRkZWQgYnkgbWFrZS5uYW1lcygpCiAgZHBseXI6Om11dGF0ZShnZW5lX3NldCA9IHN0cmluZ3I6OnN0cl9yZW1vdmUoZ2VuZV9zZXQsICJcXC4uKiIpKSAlPiUKICAjIEFkZCBhIGNvbHVtbiB0aGF0IGtlZXBzIHRyYWNrIG9mIHRoZSBnZW5lIHNldCBzaXplCiAgZHBseXI6Om11dGF0ZShnZW5lX3NldF9zaXplID0gc3RyaW5ncjo6d29yZChnZW5lX3NldCwgMiwgc2VwID0gIl8iKSkgJT4lCiAgIyBXZSB3YW50IHRvIHBsb3Qgc21hbGxlc3Qgbm8uIGdlbmVzIC0+IGxhcmdlc3Qgbm8uIGdlbmVzCiAgZHBseXI6Om11dGF0ZShnZW5lX3NldF9zaXplID0gZmFjdG9yKGdlbmVfc2V0X3NpemUsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBsZXZlbHMgPSBjKDE1LCAyNSwgNTAsIDEwMCwgMjUwLCA1MDApKSkKCmBgYAoKTGV0J3MgbWFrZSBhIHZpb2xpbiBwbG90IHNvIHdlIGNhbiBsb29rIGF0IHRoZSBkaXN0cmlidXRpb24gb2Ygc2NvcmVzIGJ5IGdlbmUgc2V0IHNpemUuCgpgYGB7ciByYW5kb21fdmlvbGlufQojIFZpb2xpbiBwbG90IGNvbXBhcmluZyBHU1ZBIHNjb3JlcyBvZiBkaWZmZXJlbnQgcmFuZG9tIGdlbmUgc2V0IHNpemVzCnJhbmRvbV9sb25nX2RmICU+JQogIGdncGxvdDI6OmdncGxvdChnZ3Bsb3QyOjphZXMoeCA9IGdlbmVfc2V0X3NpemUsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeSA9IGdzdmFfc2NvcmUpKSArCiAgIyBNYWtlIGEgdmlvbGluIHBsb3QgdGhhdCdzIGEgcHJldHR5IGJsdWUhCiAgZ2dwbG90Mjo6Z2VvbV92aW9saW4oZmlsbCA9ICIjOTlDQ0ZGIiwgYWxwaGEgPSAwLjUpICsKICAjIEFkZCBhIHBvaW50IHdpdGggdGhlIG1lYW4gdmFsdWUKICBnZ3Bsb3QyOjpzdGF0X3N1bW1hcnkoCiAgICBnZW9tID0gInBvaW50IiwKICAgIGZ1biA9ICJtZWFuIiwKICAgICMgQ2hhbmdlIHRoZSBhZXN0aGV0aWNzIG9mIHRoZSBwb2ludHMKICAgIHNpemUgPSAzLAogICAgY29sb3IgPSAiIzAwNjZDQyIsCiAgICBzaGFwZSA9IDE4CiAgKSArCiAgIyBGbGlwIHRoZSBheGVzCiAgZ2dwbG90Mjo6Y29vcmRfZmxpcCgpICsKICBnZ3Bsb3QyOjpsYWJzKHRpdGxlID0gIlJhbmRvbSBHZW5lIFNldCBHU1ZBIFNjb3JlcyIsCiAgICAgICAgICAgICAgICB4ID0gImdlbmUgc2V0IHNpemUiLAogICAgICAgICAgICAgICAgeSA9ICJHU1ZBIHNjb3JlIikgKwogIGdncGxvdDI6OnRoZW1lX2J3KCkKYGBgCgpXaGF0IGRvIHlvdSBub3RpY2UgYWJvdXQgdGhlc2UgZGlzdHJpYnV0aW9ucz8KSG93IG1pZ2h0IHlvdSB1c2UgdGhpcyBpbmZvcm1hdGlvbiB0byBpbmZvcm0geW91ciBpbnRlcnByZXRhdGlvbiBvZiBHU1ZBIHNjb3Jlcz8KCiMjIyBIb3cgY2FuIHlvdSB1c2UgdGhlc2Ugc2NvcmVzPwoKSWYgeW91IGRpZCBoYXZlIGdyb3VwcyBpbmZvcm1hdGlvbiBmb3IgeW91ciBzYW1wbGVzLCB5b3UgY291bGQgdGVzdCBmb3IgcGF0aHdheSBzY29yZXMgZGlmZmVyZW5jZXMgYmV0d2VlbiBncm91cHMuCkhlcmUncyBbYW4gZXhhbXBsZV0oaHR0cHM6Ly9odG1scHJldmlldy5naXRodWIuaW8vP2h0dHBzOi8vZ2l0aHViLmNvbS9BbGV4c0xlbW9uYWRlL09wZW5QQlRBLWFuYWx5c2lzL2Jsb2IvOWI0NGJmMWMxODZiMzEyNmIxNmRiZTViODc3NTZiM2VhZTNmZWVjMi9hbmFseXNlcy9nZW5lLXNldC1lbnJpY2htZW50LWFuYWx5c2lzLzAyLW1vZGVsLWdzZWEubmIuaHRtbCkgb2YgdGhhdCBmcm9tIHRoZSBPcGVuUEJUQSBwcm9qZWN0IGl0c2VsZiEgCgpZb3UgY2FuIGFsc28gdmlzdWFsaXplIHRoaXMgbWF0cml4IGluIGEgaGVhdG1hcC4KSGVyZSdzIGEgZmlndXJlIGZyb20gdGhlIE9wZW5QQlRBIHByb2plY3QsIHdoZXJlIHRoZSBtaWRkbGUgcGFuZWwgaXMgYSBoZWF0bWFwIG9mIEdTVkEgc2NvcmVzIHRoYXQgd2VyZSBzaWduaWZpY2FudGx5IGRpZmZlcmVudCBiZXR3ZWVuIGhpc3RvbG9naWVzLgoKIVtdKGh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9BbGV4c0xlbW9uYWRlL09wZW5QQlRBLWFuYWx5c2lzLzliNDRiZjFjMTg2YjMxMjZiMTZkYmU1Yjg3NzU2YjNlYWUzZmVlYzIvZmlndXJlcy9wbmdzL3RyYW5zY3JpcHRvbWljLW92ZXJ2aWV3LnBuZykKClsqQ29kZSBoZXJlLipdKGh0dHBzOi8vZ2l0aHViLmNvbS9BbGV4c0xlbW9uYWRlL09wZW5QQlRBLWFuYWx5c2lzL2Jsb2IvOWI0NGJmMWMxODZiMzEyNmIxNmRiZTViODc3NTZiM2VhZTNmZWVjMi9maWd1cmVzL3NjcmlwdHMvdHJhbnNjcmlwdG9taWMtb3ZlcnZpZXcuUiNMMTA2KQoKIyMjIFdyaXRlIHJlc3VsdHMgdG8gZmlsZQoKYGBge3Igd3JpdGVfZ3N2YV9yZXN1bHRzfQpnc3ZhX3Jlc3VsdHMgJT4lCiAgYXMuZGF0YS5mcmFtZSgpICU+JQogIHRpYmJsZTo6cm93bmFtZXNfdG9fY29sdW1uKCJwYXRod2F5IikgJT4lCiAgcmVhZHI6OndyaXRlX3RzdihmaWxlID0gZ3N2YV9yZXN1bHRzX2ZpbGUpCmBgYAoKIyMgU2Vzc2lvbiBJbmZvCgpgYGB7ciBzZXNzaW9uX2luZm99CnNlc3Npb25JbmZvKCkKYGBgCg==